perf: cache MatrixAccessor properties to avoid redundant recomputation by MaykThewessen · Pull Request #616 · PyPSA/linopy

MaykThewessen · 2026-03-13T21:16:14Z

Summary

Change vlabels, clabels, A, c, b, sense, lb, ub, vtypes, and Q from @property to @cached_property in MatrixAccessor
Add all new cached properties to clean_cached_properties() for proper invalidation
Add test verifying caching behavior and cache invalidation

Motivation

During to_highspy() (and other direct API exporters), MatrixAccessor properties are accessed multiple times — both directly and indirectly. For example, M.A internally accesses M.clabels and M.vlabels, and M.c accesses M.flat_vars. Each access recomputes from scratch, including rebuilding sparse matrices and flattening DataFrames.

For large models (~593K variables, ~1.38M constraints), this redundant recomputation adds measurable overhead per solve call. Caching eliminates this since the underlying model data doesn't change between property accesses within a single solve.

The cache is already properly invalidated at the start of Model.solve() and Model._mock_solve() via existing clean_cached_properties() calls.

Context

See discussion in #198 (comment) for profiling data from a real-world 52-chunk DC-OPF optimization.

Test plan

Existing test_matrices.py tests pass (verify shapes, values, masked models)
New test_matrices_properties_are_cached verifies identity caching and invalidation
test_optimization.py highs-direct tests pass (24/25 — one pre-existing failure in test_modified_model)
test_io.py tests pass

Note: test_modified_model fails on upstream master as well — it's a pre-existing issue unrelated to this change.

🤖 Generated with Claude Code

Covers the code paths optimised by these PRs: - PyPSA#616 cached_property on MatrixAccessor (flat_vars / flat_cons) - PyPSA#617 np.char.add for label string concatenation - PyPSA#618 sparse matrix slicing in MatrixAccessor.A - PyPSA#619 numpy solution unpacking Reproduces benchmark results on PyPSA SciGrid-DE (24–500 snapshots) and a synthetic model. Supports JSON output and --compare mode for cross-branch comparison. Reproduce with: python benchmark/scripts/benchmark_matrix_gen.py -o results.json --label "after" Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Reproduces the performance claims for PRs PyPSA#616–PyPSA#619 on PyPSA SciGrid-DE and a synthetic model. python benchmark/scripts/benchmark_matrix_gen.py -o results.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

MaykThewessen · 2026-03-17T20:25:05Z

Added benchmark/scripts/benchmark_matrix_gen.py to this branch (and #617, #618, #619) as requested by @FBumann.

Reproduce with:

python benchmark/scripts/benchmark_matrix_gen.py -o results.json --label "with-PR-616"
python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json

The benchmark times flat_vars, flat_cons, vlabels+clabels, A_matrix, and the full get_matrix_data pipeline on PyPSA SciGrid-DE (24–500 snapshots) and a synthetic model. This PR's caching benefit is most visible when the pipeline is called multiple times per solve (3–4× in the HiGHS direct path), which the script's full_pipeline phase reflects via clean_cached_properties() + re-run.

Adds benchmark/scripts/benchmark_matrix_gen.py covering all four performance code paths: - PyPSA#616 cached_property on MatrixAccessor (flat_vars / flat_cons) - PyPSA#617 np.char.add label string concatenation - PyPSA#618 single-step sparse matrix slicing - PyPSA#619 numpy dense-array solution unpacking Reproduce with: python benchmark/scripts/benchmark_matrix_gen.py -o results.json python benchmark/scripts/benchmark_matrix_gen.py --include-solve # PR PyPSA#619 python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Adds benchmark/scripts/benchmark_matrix_gen.py covering all four performance code paths: - PyPSA#616 cached_property on MatrixAccessor (flat_vars / flat_cons) - PyPSA#617 np.char.add label string concatenation - PyPSA#618 single-step sparse matrix slicing - PyPSA#619 numpy dense-array solution unpacking Reproduce with: python benchmark/scripts/benchmark_matrix_gen.py -o results.json python benchmark/scripts/benchmark_matrix_gen.py --include-solve # PR PyPSA#619 path python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Adds benchmark/scripts/benchmark_matrix_gen.py covering all four performance code paths: - PyPSA#616 cached_property on MatrixAccessor (flat_vars / flat_cons) - PyPSA#617 np.char.add label string concatenation - PyPSA#618 single-step sparse matrix slicing - PyPSA#619 numpy dense-array solution unpacking Reproduce with: python benchmark/scripts/benchmark_matrix_gen.py -o results.json python benchmark/scripts/benchmark_matrix_gen.py --include-solve # PR PyPSA#619 python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Converts vlabels, vtypes, lb, ub, clabels, A, sense, b, c, Q from @Property to @cached_property, and extends clean_cached_properties() to clear all of them. Avoids recomputing expensive matrix operations (e.g. flat DataFrame flattening, sparse matrix slicing) on each access.

Adds benchmark/scripts/benchmark_matrix_gen.py covering all four performance code paths: - PyPSA#616 cached_property on MatrixAccessor (flat_vars / flat_cons) - PyPSA#617 np.char.add label string concatenation - PyPSA#618 single-step sparse matrix slicing - PyPSA#619 numpy dense-array solution unpacking Reproduce with: python benchmark/scripts/benchmark_matrix_gen.py -o results.json python benchmark/scripts/benchmark_matrix_gen.py --include-solve # PR PyPSA#619 python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

for more information, see https://pre-commit.ci

FBumann · 2026-03-18T07:23:06Z

@MaykThewessen Can you provide the results you got with the benchmarks? Compare master vs feature branch.
Its best to see your results (summary table), and then see if we can replicate them if needed. This also serves as permanent documentation on the PR. Also applies to #617 #618 #619
Also in general: The benchmark should test the actual implementation, not a mock up inside the benchmark file if possible. This ensures that the benchmark captures side effects.

FBumann · 2026-03-18T07:47:53Z

@MaykThewessen THis PR in general looks good. We need to take care with cache cleanup though. Ill look into it.

FBumann · 2026-03-18T08:56:17Z

If you want you can share your thoughts on #622, which will hopefully cover your workflow even better.

MaykThewessen · 2026-03-18T09:22:52Z

Benchmark Results: master vs PR #616

Tested on actual linopy implementation (not mocked) using PyPSA SciGrid-DE at multiple problem sizes.
Each phase is called directly on model.matrices — the real code path used by solvers.

Setup: Python 3.14.3, numpy 2.4.3, Apple M-series (arm64), macOS, 5 repeats (best-of).

Model	Snapshots	Phase	master (s)	PR-616 (s)	Speedup
scigrid	24	flat_vars	0.0055	0.0051	1.08x
scigrid	24	flat_cons	0.1510	0.1359	1.11x
scigrid	24	vlabels+clabels	0.1606	0.1378	1.17x
scigrid	24	A_matrix	0.1649	0.1363	1.21x
scigrid	24	full_matrix_pipeline	0.3494	0.2973	1.18x
scigrid	100	flat_vars	0.0141	0.0123	1.14x
scigrid	100	flat_cons	0.7305	0.5268	1.39x
scigrid	100	vlabels+clabels	0.6693	0.5878	1.14x
scigrid	100	A_matrix	0.7467	0.6273	1.19x
scigrid	100	full_matrix_pipeline	2.0155	1.2550	1.61x
scigrid	200	flat_vars	0.0231	0.0192	1.20x
scigrid	200	flat_cons	2.2360	1.3243	1.69x
scigrid	200	vlabels+clabels	2.1735	1.2356	1.76x
scigrid	200	A_matrix	1.7033	1.1936	1.43x
scigrid	200	full_matrix_pipeline	3.2036	2.5709	1.25x
scigrid	500	flat_vars	0.0449	0.0446	1.01x
scigrid	500	flat_cons	5.9889	5.0979	1.17x
scigrid	500	vlabels+clabels	5.6156	5.1236	1.10x
scigrid	500	A_matrix	5.3982	5.3102	1.02x
scigrid	500	full_matrix_pipeline	11.6722	10.5574	1.11x

Summary: Consistent 1.1–1.8x speedup across matrix generation phases, with larger improvements at medium problem sizes (100–200 snapshots). The caching benefit is most visible on flat_cons and vlabels+clabels where redundant recomputation is eliminated.

Benchmark methodology

Each phase calls the actual model.matrices property (e.g., matrices.flat_vars, matrices.A)
Cache is cleared with matrices.clean_cached_properties() before each measurement
full_matrix_pipeline accesses all properties solvers use: vlabels, clabels, lb, ub, A, b, c, sense
5 repeats per measurement, best-of-5 reported
GC disabled during timing, collected between repeats
Benchmark script: benchmark/scripts/benchmark_actual.py

MaykThewessen · 2026-03-18T09:23:52Z

@FBumann Thanks for reviewing! Happy to help with the cache cleanup — let me know if you'd like me to look at specific scenarios where the cache needs invalidation, or if you'd prefer to handle it yourself.

FBumann · 2026-03-18T10:46:14Z

@MaykThewessen Thinking about those scenarios is the actual work. So if you can, a list of such would help. The actual implementation is easy then.

FBumann · 2026-03-18T10:50:46Z

@MaykThewessen As is said in #619, your Benchmark does not really isolate the Content if this PR. And its not properly measuring the caching improvement, as you call clear_cached_properties in places where you should not...

I can merge this anyway as im convinced that its an improvement, but please take your time and check that your benchmark actually measures what its supposed to when publishing results next time.

MaykThewessen · 2026-03-18T11:55:19Z

You're right about the benchmark — calling clean_cached_properties() before each measurement measures the uncached path every time, defeating the purpose of testing caching. Will be more careful with methodology next time.

On cache invalidation scenarios:

Mutation	Properties to invalidate
`model.add_variables()` / `remove_variables()`	flat_vars, vlabels, lb, ub, vtypes, c, A, Q
`model.add_constraints()` / `remove_constraints()`	flat_cons, clabels, A, b, sense
Variable bounds modified (`var.lower = ...`, `var.upper = ...`)	lb, ub
Constraint RHS modified (`con.rhs = ...`)	b
Constraint sign modified	sense
Objective coefficients modified (`model.add_objective()`)	c, Q
Constraint coefficients modified (if supported in-place)	A

The first two (add/remove variables/constraints) are the critical ones since they change the shape of the matrix. Bounds/RHS/objective modifications only change values within an existing structure.

One approach: blanket clean_cached_properties() on any structural mutation (add/remove), and targeted per-property invalidation for value-only changes. Or just blanket-invalidate everything on any mutation — simpler and safe given these properties are cheap relative to solver time.

MaykThewessen mentioned this pull request Mar 13, 2026

perf: replace np.vectorize with vectorized string ops for label names #617

Closed

3 tasks

FBumann mentioned this pull request Mar 14, 2026

chore: benchmarks #567

Open

3 tasks

FBumann added the performance label Mar 17, 2026

This was referenced Mar 17, 2026

perf: use single-step sparse matrix slicing in MatrixAccessor #618

Closed

perf: use numpy array lookup for solution unpacking #619

Open

MaykThewessen force-pushed the perf/cache-matrix-accessor-properties branch from 3373a1b to bc3b49e Compare March 17, 2026 20:38

MaykThewessen force-pushed the perf/cache-matrix-accessor-properties branch from f499edd to e639bb6 Compare March 17, 2026 20:57

MaykThewessen and others added 2 commits March 17, 2026 22:13

MaykThewessen force-pushed the perf/cache-matrix-accessor-properties branch from 9f21863 to 4985f9a Compare March 17, 2026 21:13

[pre-commit.ci] auto fixes from pre-commit.com hooks

3685512

for more information, see https://pre-commit.ci

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: cache MatrixAccessor properties to avoid redundant recomputation#616

perf: cache MatrixAccessor properties to avoid redundant recomputation#616
MaykThewessen wants to merge 3 commits intoPyPSA:masterfrom
MaykThewessen:perf/cache-matrix-accessor-properties

MaykThewessen commented Mar 13, 2026

Uh oh!

MaykThewessen commented Mar 17, 2026

Uh oh!

FBumann commented Mar 18, 2026

Uh oh!

FBumann commented Mar 18, 2026

Uh oh!

FBumann commented Mar 18, 2026

Uh oh!

MaykThewessen commented Mar 18, 2026

Uh oh!

MaykThewessen commented Mar 18, 2026

Uh oh!

FBumann commented Mar 18, 2026

Uh oh!

FBumann commented Mar 18, 2026

Uh oh!

MaykThewessen commented Mar 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

MaykThewessen commented Mar 13, 2026

Summary

Motivation

Context

Test plan

Uh oh!

MaykThewessen commented Mar 17, 2026

Uh oh!

FBumann commented Mar 18, 2026

Uh oh!

FBumann commented Mar 18, 2026

Uh oh!

FBumann commented Mar 18, 2026

Uh oh!

MaykThewessen commented Mar 18, 2026

Benchmark Results: master vs PR #616

Uh oh!

MaykThewessen commented Mar 18, 2026

Uh oh!

FBumann commented Mar 18, 2026

Uh oh!

FBumann commented Mar 18, 2026

Uh oh!

MaykThewessen commented Mar 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants